The validity of the selection methods for recruitment to UK core psychiatry training: cohort study.

AIMS AND METHOD: Selection into core psychiatry training in the UK uses a computer-delivered Multi-Specialty Recruitment Assessment (MSRA; a situational judgement and clinical problem-solving test) and, previously, a face-to-face Selection Centre. The Selection Centre assessments were suspended during the COVID-19 pandemic. We aimed to evaluate the validity of this selection process using data on 3510 psychiatry applicants. We modelled the ability of the selection scores to predict subsequent performance in the Clinical Assessment of Skills and Competencies (CASC). Sensitivity to demographic characteristics was also estimated. RESULTS: All selection assessment scores demonstrated positive, statistically significant, independent relationships with CASC performance and were sensitive to demographic factors. IMPLICATIONS: All selection components showed independent predictive validity. Re-instituting the Selection Centre assessments could be considered, although the costs, potential advantages and disadvantages should be weighed carefully.

New evidence on the validity of the selection methods for recruitment to general practice training: a cohort study.

BACKGROUND: Selection into UK-based GP training has used the multi-specialty recruitment assessment (MSRA) and a face-to-face selection centre. The MSRA comprises of a situational judgement test (SJT) and clinical problem solving (CPS) test. The selection centre was suspended during the Covid-19 pandemic. Evidence is needed to guide national and international selection policy. AIM: To evaluate the validity of GP training selection. DESIGN & SETTING: A retrospective cohort study using data from UK-based national recruitment to GP training, from 2015 to 2021. METHODS: Data were available for 32,215 GP training applicants. The ability of scores from the specialty selection process to predict subsequent performance in the clinical skills assessment of the Membership of the Royal College of General Practitioners (MRCGP) examination was modelled using path analysis. The effect-sizes for gender, professional family background and world region of qualification were estimated. RESULTS: All component scores of the selection process demonstrated statistically significant independent relationships with clinical skills assessment performance, thus establishing their predictive validity. All were sensitive to demographic factors. The selection centre scores had the weakest relationship with future clinical skills assessment performance. However, for candidates with MSRA scores below the lowest quartile the relative contribution of the selection centre scores to predicting clinical skills assessment performance was similar to that observed for MSRA components. CONCLUSION: The MSRA has predictive validity in this context. Re-instituting a selection centre for those with relatively low MSRA scores should be considered. However, the relative costs and potential advantages and disadvantages should be carefully weighed.

Demographic disparities in dental school selection: An analysis of current UK practices.

INTRODUCTION: Specific social groups remain under-represented within dentistry. While the University Clinical Aptitude Test (UCAT) aims to widen participation in under-represented social groups, there is no evidence in dental education that this aim is being met. MATERIALS AND METHODS: Data over two admission cycles (2012 and 2013), including 3246 applicants to 10 UK dental schools, were analysed. Applicant and selected pools were compared to the UK population. Multiple logistic regression was used to investigate the association between demographic variables and UCAT and receiving an offer of a place at dental school. RESULTS: Over-representation of Female, Asian, least deprived and grammar school groups were found in applicant and selected pools compared with the UK population. White ethnic applicants were significantly more selected than Black (OR 0.25), Asian (OR 0.57) and Mixed (OR 0.80) ethnicities, while least deprived applicants were significantly more selected than most deprived (OR 0.59). Grammar school education increased odds of selection by 1.8 when compared to state school. The addition of UCAT to the model for applicants reduced ethnic disparities but led to disparities between other groups. CONCLUSION: Current widening participation practices focus on attracting applicants from lower socio-economic groups. However, this study showed that ethnicity, sex and educational background biases also affect demographic diversity in dentistry. The UCAT shows promise in levelling the playing field; however, widening access measures will only succeed if selection committees radically change selection processes to address the systemic biases, enabling the dentists of tomorrow to represent the society they serve.

Evidence-based appraisal of the role of SJTs in selection.

A recent opinion article in Clinical Medicine promoted a new preference-based algorithm to allocate training places for the UK Foundation Programme Office (UKFPO). This replaced the previous process, which ranked candidates based on medical school academic achievement (the educational performance measure; EPM) and the score on a situational judgement test (SJT). Although not without risks, we believe that the new system has positive potential. In presenting their case, Sam et al summarised evidence relating to the UKFPO in an unbalanced way, leading to what we believe are erroneous inferences, particularly with regard to differential attainment. Here, we provide an example of how the general evidence base and conceptual understanding of the validity of SJTs for medical selection is poorly understood. We highlight important research findings that were not cited by Sam et al and provide what we believe is a more balanced and accurate interpretation of the evidence base relating the UKFPO SJT, and SJTs used in medical selection in general. We do this with particular reference to the validity of such tools in this context, as well as their potential impact on under-represented groups in medicine, compared with other selection assessments.

Evaluation of Situational Judgment Tests in student selection in Indonesia and the impact on diversity issues.

BACKGROUND: Internationally, medical selection relies heavily on prior academic attainment which has an adverse impact on the diversity of selected students. Since non-academic attributes are also important, this study aims to evaluate the use of a Situational Judgment Test (SJT) for selection and the impact on student diversity relating to gender, ethnicity and socio-economic status. Previous SJT research has almost entirely originated from a Western context and this study focuses on new evidence in a South East Asian context with a different demographic profile. METHODS: Thirty faculty members developed 112 SJT scenarios assessing professionalism, communication and self-awareness domains. The scenarios underwent a concordance stage where stakeholder input was sought on the content appropriateness, to define the item scoring key, followed by an initial psychometric evaluation with first and second year medical students (N = 436). Based on these results, 30 scenarios, consisting of 128 nested items, were selected for pilot testing and evaluation regarding diversity issues with two cohorts of applicants in 2017 (N = 446) and 2018 (N = 508). RESULTS: The SJT demonstrated good internal consistency (Cronbach's alpha of 0.80 and 0.81 respectively). There were significant differences in SJT scores based on gender in both years, where females consistently outperformed males (p = .0001). However, no significant differences were found based on high school origin, parental educational background or ethnicity. CONCLUSIONS: This is the first study to evaluate the use of an SJT in Indonesia, which has a unique diversity profile compared to Western countries. Largely, the preliminary results replicate previous studies of the potential diversity benefits of using an SJT as a tool for medical student selection and has the potential to level the playing field regarding socio-economic status and ethnicity. Further studies exploring more variables representing diversity are warranted to confirm the early results in this study.

Development and Validation of a Situational Judgement Test to Assess Professionalism.

Objective. The purpose of this study was to develop, pilot, and validate a situational judgement test (SJT) to assess professionalism in Doctor of Pharmacy (PharmD) students. Methods. Test specifications were developed and faculty members were educated on best practices in item writing for SJTs. The faculty members then developed 75 pilot scenarios. From those, two versions of the SJT, each containing 25 scenarios, were created. The pilot population for the SJT was student pharmacists in their third professional year, just prior to starting their advanced pharmacy practice experiences. The students completed the two versions of the test on different days, approximately 48 hours apart, with 50 minutes allowed to complete each. Subsequently, students completed a questionnaire regarding the SJT at the conclusion of the second test. Results. Version 1 of the SJT was completed by 228 students, and version 2 was completed by 225 students. Mean scores were 390 (SD=20, range 318-429) and 342 (SD=21, range 263-387) on test versions 1 and 2, respectively. The reliability of the tests was appropriate (test version 1, α=0.77; test version 2, α=0.79). Students felt that the content of the tests was realistic with respect to pharmacy practice (90.1%), and that the tests gave them an opportunity to reflect on how to approach challenging situations (82.6%). Conclusion. We developed a reliable SJT to assess professionalism in PharmD students. Future research should focus on creating a personalized learning plan for students who do not meet minimum performance standards on this SJT.

Evaluation of a Situational Judgement Test to Develop Non-Academic Skills in Pharmacy Students.

Objective. To design, implement, and psychometrically evaluate a situational judgement test (SJT) to use as a formative assessment of pharmacy students' non-academic skills in an Australian-based university. Methods. An SJT was developed using a previously validated design process including involvement of subject matter experts. The first phase included design of a blueprint through stakeholder consultation and the development of bespoke attribute definitions and a tool specification. Following on from this, SJT items were developed through subject matter expert interviews and in-depth review process. Results. Students (702) from four different cohorts (first through fourth years) of a Bachelor of Pharmacy degree program completed the situational judgment test. Data from 648 students was eligible for inclusion in the analysis. The SJT demonstrated good reliability, appropriateness for use (difficulty and quality), fairness, and face validity. The variability in students' scores suggested that the SJT may be a useful metric to identify students most in need of additional support. Conclusion. Evaluation of the SJT demonstrated that the tool was valid, reliable, fair and appropriate to use as a formative assessment. Through implementing an SJT such as this, pharmacy students are provided the opportunity to receive feedback on their non-academic skills and consider how to approach challenging or unfamiliar situations before entering the profession.

A literature review of the predictive validity of European dental school selection methods.

INTRODUCTION: Selection to dental school is the point at which there is the potential to assess a wide range of candidate attributes and select those most likely to learn, train and work within the profession. Despite this, little is known in terms of what works and what does not work in dental selection in terms of predicting future performance accurately and fairly. Given this, our aim was to synthesise the last 30 years of research investigating the predictive validity of dental school selection methods. METHODS: A search of the electronic databases SCOPUS, Pubmed and Embase was conducted. Results were limited to English language studies published between January 1987 and January 2017. RESULTS: Twenty-one studies were included. Selection tools fell into five broad categories: tests of personal qualities; cognitive ability; academic attainment; psychomotor skills and combined ability tests. Most were retrospective, single-site studies limited to early years of dental school. Weak correlations were reported, but in most cases, these were between small sections of the selection tool and/or the outcome measure. DISCUSSION: There was a notable dearth of published research examining dental schools selection processes across Europe over the last 30 years. Current literature was limited by weak study design and lack of long-term follow-up. CONCLUSION: There is insufficient high-quality evidence from which to draw any conclusions as to the best selection methods to use in dental school selection. Without this, designing selection frameworks for dentistry which are appropriately weighted, reliable and valid remains a challenge.

Do changing medical admissions practices in the UK impact on who is admitted? An interrupted time series analysis.

INTRODUCTION: Medical admissions must balance two potentially competing missions: to select those who will be successful medical students and clinicians and to increase the diversity of the medical school population and workforce. Many countries address this dilemma by reducing the heavy reliance on prior educational attainment, complementing this with other selection tools. However, evidence to what extent this shift in practice has actually widened access is conflicting. AIM: To examine if changes in medical school selection processes significantly impact on the composition of the student population. DESIGN AND SETTING: Observational study of medical students from 18 UK 5-year medical programmes who took the UK Clinical Aptitude Test from 2007 to 2014; detailed analysis on four schools. PRIMARY OUTCOME: Proportion of admissions to medical school for four target groups (lower socioeconomic classes, non-selective schooling, non-white and male). DATA ANALYSIS: Interrupted time-series framework with segmented regression was used to identify the impact of changes in selection practices in relation to invitation to interview to medical school. Four case study medical schools were used looking at admissions within for the four target groups. RESULTS: There were no obvious changes in the overall proportion of admissions from each target group over the 8-year period, averaging at 3.3% lower socioeconomic group, 51.5% non-selective school, 30.5% non-white and 43.8% male. Each case study school changed their selection practice in decision making for invite to interview during 2007-2014. Yet, this within-school variation made little difference locally, and changes in admission practices did not lead to any discernible change in the demography of those accepted into medical school. CONCLUSION: Although our case schools changed their selection procedures, these changes did not lead to any observable differences in their student populations. Increasing the diversity of medical students, and hence the medical profession, may require different, perhaps more radical, approaches to selection.

Thinking of selection and widening access as complex and wicked problems.

OBJECTIVES: 'Wicked problems' are complex in nature, have innumerable causes associated with multiple social environments and actors with unpredictable behaviour and outcomes, and are difficult to define or even resolve. This paper considers why and how the frameworks of complexity theory and wicked problems can help medical educators consider selection and widening access (WA) to medicine through fresh eyes to guide future policy and practice. We illustrate how 'wickedity' can frame the key issues in this area, and then address steps that education stakeholders might take to respond to and act on these issues. METHODS: We used the 10 properties of a wicked problem to frame common issues in the broad field of selection and WA in medicine. We drew heavily on literature from different disciplines, particularly education, and, through debate and reflection, agreed on the applicability of the theory for illuminating and potentially addressing outstanding issues in selection and WA. RESULTS: Framing medical school selection using the 10 properties of wicked problems is a means of shifting thinking from erroneous 'simple' solutions to thinking more contextually and receptively. The wicked problem framework positions selection as a multi-causal, complex, dynamic, social problem and foregrounds stakeholders' views and context as being highly relevant in medical school selection. CONCLUSIONS: The wicked problem lens shifts thinking and action from seeking one elusive, objective truth to recognising the complexity of medical school selection, managing uncertainty, questioning and considering 'issues' associated with medical school selection more productively. Although there are criticisms of this framework, labelling medical selection as 'wicked' provides original insights and genuine reframing of the challenges of this important, and high profile, aspect of medical education. Doing so, in turn, opens the door to different responses than would be the case if selection and WA were simple and readily tamed.

Unpacking the dark variance of differential attainment on examinations in overseas graduates.

CONTEXT: Differential performance in postgraduate examinations between home medical graduates and those who qualified outside their country of practice is well recognised. This difference is especially marked in the practical component of the UK Membership of the Royal College of General Practitioners (MRCGP) examination. The potential causes of such disparities are not well understood. METHODS: Data were available for 1874 international medical graduates who applied for general practice (GP) specialty training in the UK in 2008-2012. The primary outcome was performance in the Clinical Skills Assessment (CSA) OSCE component of the MRCGP. The main predictors were performance on a situational judgement test (SJT) and clinical problem-solving test (CPST), a test of applied clinical knowledge, used in the selection for GP training. Data relating to demographic characteristics and English language fluency were also available. To better understand the relationship between the predictors, the selection measures and the outcome, a series of univariable and multivariable models were developed and tested, concluding with a structural equation model to explore causality. RESULTS: The CSA rating was more strongly predicted by SJT scores (standardised beta, 0.26) than by performance on the CPST (standardised beta, 0.17). There was a relationship between English language fluency and CSA score that was mainly mediated via SJT performance. CONCLUSIONS: These findings demonstrate that performance on an SJT predicts performance in a high-fidelity clinical simulation (the CSA) in international medical graduates. Although the constructs tested by SJTs are debated, and are likely to vary across settings, culturally appropriate knowledge of interpersonal competence is likely to be evaluated. Improving the confidence of doctors in this area through targeted educational interventions, rather than focusing on increased clinical knowledge, is likely to be more effective at reducing disparities observed in postgraduate examination performance. Thus, there are important implications for the design of specialty selection and licensing assessments globally.

Opening Editorial: Selection and Recruitment in Medical Education.

This article was migrated. The article was marked as recommended. There is over a century of research on selection and recruitment and the field has both developed and expanded significantly over this time. Previous research has tended to focus on reviewing the effectiveness of selection methods (academic records, references, personal statements, aptitude tests, personality assessments, situational judgement tests, and interviews), where good quality evidence is now emerging. Many challenges remain however, reflecting that selection and recruitment into medical education (both undergraduate and postgraduate) is a complex, multi-dimensional, dynamic phenomenon. For example, issues regarding diversity and fairness in selection have been researched over many years but there remains a huge gap between the research evidence and policy enactment in many parts of the globe. In this opening editorial for our special issue on selection and recruitment in medical education we encourage authors to consider six key question areas (amongst others), including: •how will technology (e.g. social media, big data, artificial intelligence, etc) influence selection research and practices in future?•should selection criteria be reviewed to include creativity, innovation, resilience and adaptability (beyond heavy reliance on prior academic attainment as the main criterion)?•is selection for medical education fair? How do we address issues regarding widening participation and diversity in practice?•to what extent do political, cultural and social factors influence selection philosophy and policies internationally?•what are the risks to effective selection (e.g. access to coaching, legal challenge of poor practices) and,•a new Ottawa consensus on selection and recruitment has been published - to what extent does this statement reflect your experiences of designing and implementing selection systems in your locality? In contributing to the debate, this special issue provides a platform for authors to present the latest research, empirical studies, systematic reviews, reflections, case studies and practical tips on current/future issues in selection and recruitment in medical education.

An exploration of applicant perceptions of asynchronous video MMIs in medical selection.

This article was migrated. The article was marked as recommended. Over the last two decades, technological advancements internationally have meant that the Internet has become an important medium for recruitment and selection. Consequently, there is an increased need for research that examines the effectiveness of newer technology-mediated selection methods. This exploratory research study qualitatively explored applicant perceptions of fairness of asynchronous video interviews used in medical selection. Ten undergraduate medical students participated in a pilot asynchronous multiple-mini interview and were invited to share their experiences and perceptions in a follow-up interview. The data was transcribed verbatim and analysed using template analysis, with Gilliland's (1993) organisational justice theory guiding the original template. Many of the original themes from Gilliland's model were uncovered during analysis. Additionally, some significant themes were identified that did not form part of the original template and were therefore added to the final coding template - these were specifically relating to technology, including acceptability in a medical context; technical issues and adverse impact. Overall, results suggested that participants perceived asynchronous video interviews to be a fair method of selection. However, participants thought asynchronous interviews should only be used as part of an extensive selection process and furthermore, should not replace face-to-face interviews. Findings are discussed in line with existing research of fairness perceptions and justice theory in selection ( Gilliland, 1993) and implications for research and practice are presented.

Closing Editorial: New insights and reflections on the science of selection and recruitment.

This article was migrated. The article was marked as recommended. In this closing editorial we reflect on key topics presented in this special issue on selection and recruitment, with a view to identifying gaps in the literature, and exploring where next. Four key themes have emerged including; (1) the impact of using new technologies in selection and recruitment; (2) addressing social accountability, diversity and fairness issues in selection; (3) increased emphasis on non-academic personal attributes in selection, and (4) attraction and recruitment in postgraduate recruitment. The implications of findings from this collection of studies and opinion pieces are discussed in relation to future research, policy and practice.

A preliminary investigation to explore the cognitive resources of physicians experiencing difficulty in training.

BACKGROUND: Treating patients is complex, and research shows that there are differences in cognitive resources between physicians who experience difficulties, and those who do not. It is possible that differences in some cognitive resources could explain the difficulties faced by some physicians. In this study, we explore differences in cognitive resources between different groups of physicians (that is, between native (UK) physicians and International Medical Graduates (IMG); those who continue with training versus those who were subsequently removed from the training programme); and also between physicians experiencing difficulties compared with the general population. METHODS: A secondary evaluation was conducted on an anonymised dataset provided by the East Midlands Professional Support Unit (PSU). One hundred and twenty one postgraduate trainee physicians took part in an Educational Psychology assessment through PSU. Referrals to the PSU were mainly on the basis of problems with exam progression and difficulties in communication skills, organisation and confidence. Cognitive resources were assessed using the Wechsler Adult Intelligence Scale (WAIS-IV). Physicians were categorised into three PSU outcomes: 'Continued in training', 'Removed from training' and 'Active' (currently accessing the PSU). RESULTS: Using a one-sample Z test, we compared the referred physician sample to a UK general population sample on the WAIS-IV and found the referred sample significantly higher in Verbal Comprehension (VCI; z = 8.78) and significantly lower in Working Memory (WMI; z = -4.59). In addition, the native sample were significantly higher in Verbal Comprehension than the UK general population sample (VCI; native physicians: z = 9.95, p < .001, d = 1.25), whilst there was a lesser effect for the difference between the IMG sample and the UK general population (z = 2.13, p = .03, d = 0.29). Findings also showed a significant difference in VCI scores between those physicians who were 'Removed from training' and those who 'Continued in training'. CONCLUSIONS: Our results suggest it is important to understand the cognitive resources of physicians to provide a more focussed explanation of those who experience difficulties in training. This will help to implement more targeted interventions to help physicians develop compensatory strategies.